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SUMMARY 


The  Air  Force  Research  Laboratory,  Warfighter  Training  Research  Division 
(AFRL/HEA),  is  conducting  research  on  the  technologies  required  to  develop  and  support 
Distributed  Mission  Training  (DMT)  and  on  training  applications  and  strategies  for  DMT.  The 
objectives  of  training  effectiveness  research  are  to  identify  high-payoff  applications,  develop 
prototype  training  strategies,  and  assess  the  impact  of  DMT  experience.  This  report  describes  a 
research  effort  assessing  the  application  of  DMT  to  Flight  Lead  Upgrade  (FLUG)  training  for 
F-16  pilots. 

In  FLUG  training,  pilots  learn  the  skills  required  to  lead  flights  of  two  and  four  F-16s. 
These  skills  include  mission  planning,  briefing,  debriefing,  and  flight  leadership  (situation 
awareness,  tactical  execution,  work  with  air  weapons  controllers,  and  communication).  FLUG 
training  can  be  resource  intensive  requiring  both  friendly  and  opposing  forces.  Repeating 
training  flights  because  an  upgrading  FLUG  pilot  fails  to  demonstrate  acceptable  proficiency 
places  significant  demands  on  squadron  resources.  The  goal  of  this  research  program  was  to 
determine  whether  DMT  experience  would  enhance  pilot  performance  and  reduce  the  number  of 
repeated  FLUG  training  missions.  AFRL/HEA  researchers  worked  with  wing  and  squadron 
leaders  at  the  27'*’  Eighter  Wing,  Cannon  AEB  NM,  and  identified  four  vs.  four  Dissimilar  Air 
Combat  Tactics  (DACT)  and  four-ship  Surface  Attack  Tactics  (SAT)  as  potential  high-payoff 
missions  for  DMT.  A  five-day  DMT-EEUG  training  protocol  was  developed  to  enhance  pilot 
.skills  for  these  missions.  Between  June  1999  and  Eebruary  2000,  twelve  upgrading  EEUG  pilots 
supported  by  wingmen,  instructor  pilots,  and  air  weapons  controllers  participated  in  DMT 
research  exercises  at  AERE/HEA  using  this  training  protocol.  Pilot  performance  within  DMT 
missions  and  on  subsequent  aircraft  training  flights  together  with  feedback  from  upgrading  pilots 
and  their  instructors  indicate  that  DMT  experience  improves  both  individual  pilot  skills  for  tasks 
that  are  rarely  practiced  in  the  aircraft  and  leadership  skills. 
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USING  DISTRIBUTED  MISSION  TRAINING 
TO  AUGMENT  ELIGHT  LEAD  UPGRADE  TRAINING 


INTRODUCTION 

The  Air  Force’s  Distributed  Mission  Training  (DMT)  program  is  a  major  advance  in 
ground-based  training  that  will  allow  pilots  and  other  warfighters  to  train  for  complex, 
multiplayer  combat  operations  using  a  network  of  flight  simulators  and  other  systems.  Chapman 
(1998)  states  that, 

The  DMT  network  will  be  created  incrementally,  focusing  first  on  team  training. 

F-15C  and  AW ACS  Missions  Training  Centers  (MTCs)  will  be  the  first  DMT 
constituents.  MTCs  will  house  the  unit’s  simulators  or  mission  training  system, 
briefing  and  debriefing  systems,  threat  stations  and  simulations,  and  the 
infrastructure  needed  for  local  and  wide  area  networking,  (p.  1) 

F-15  and  AW ACS  MTCs  went  into  service  in  2000,  and  the  first  F-16  MTC  will  be  in  service  by 

2002. 

Researchers  from  the  Air  Force  Research  Laboratory,  Warfighter  Training  Research  Division 
(AFRL/HEA),  are  investigating  strategies  for  incorporating  DMT  into  advanced  flying  training 
in  operational  units.  When  reviewing  the  utility  of  an  earlier  generation  of  training  systems, 
Polzella,  Hubbard,  Brown,  and  McLean  (1987)  discovered  that  many  of  the  instructional  features 
that  had  been  incorporated  into  Air  Force  flight  training  simulators  were  unwanted  and  unused. 
A  primary  goal  of  DMT  research  at  AFRL/HEA  is  to  provide  commanders  and  instructors  at 
MTCs  with  effective  and  validated  training  tools  and  materials  including  training  protocols  to 
meet  specific  needs,  sample  scenarios,  and  performance  assessment  instruments.  The 
laboratory’s  objective  is  to  assist  Air  Eorce  warfighters  use  DMT  systems  to  effectively  and 
efficiently  augment  flying  training. 

Background 

Research  on  training  effectiveness  of  multiship  simulation  systems  has  been  ongoing  at 
AERE/HEA  for  almost  ten  years.  In  addition  to  research  on  networked  simulation  technologies 
activities  have  focused  on  application  of  DMT  for  continuation  training  of  fighter  pilots  and  air 
weapons  controllers.  Hoog  (1999)  described  how  DMT  systems  are  being  designed  to  enhance 
skills  that  warfighters  will  employ  both  as  individuals  and  as  members  of  a  team.  Effective 
application  of  multiplayer  simulation  for  enhancing  individual  and  team  skills  has  been 
demonstrated  for  E-15  pilots  (Houck,  Thomas,  &  Bell,  1991),  E-16  pilots  (Berger  &  Crane, 
1993),  Tornado  pilots  and  navigators  (Huddlestone,  Harris,  &  Tinsworth,  1999),  pilots,  forward 
air  controllers,  and  ground  forces  executing  close  air  support  (Bell,  et  ah,  1996),  and  Air  Eorce 
Special  Operations  teams  (Nullmeyer  &  Spiker,  in  press).  E-16  pilots  who  have  flown  in  a 
distributed  environment  have  rated  DMT  as  particularly  effective  for  training  4-ship  air-to-air 
employment  against  multiple  enemy  aircraft  (Crane,  Schiflett,  &  Oser,  2000).  E-16  pilots  have 
identified  individual  skills  including  radar  mechanization  (i.e.,  using  the  various  modes  and 
capabilities  of  the  air-to-air  radar  to  detect,  track,  and  target  multiple  aircraft),  communication. 


1 


and  building  situation  awareness  as  being  enhanced  by  DMT.  Team  skills  enhanced  by  DMT 
include  maintaining  mutual  support,  tactical  execution,  and  flight  leadership.  Crane  (1999) 
reported  that  F-16  instructor  pilots  participating  in  DMT  research  would  frequently  fly  as  flight 
leader  for  the  morning  mission  but  would  then  assign  a  less-experienced  pilot  to  serve  as  flight 
lead  during  the  afternoon.  The  instructors  explained  that  these  pilots  were  participating  in  Flight 
Lead  Upgrade  (FLUG)  training  at  their  squadrons  and  that  DMT  was  a  valuable  complement  to 
aircraft  training. 

Flight  Lead  Upgrade  Training 

After  graduation  from  a  formal  training  course,  U.  S.  Air  Force  fighter  pilots  must  first 
complete  Mission  Qualification  Training  at  their  assigned  unit  before  serving  as  mission-ready 
wingmen.  After  gaining  the  required  flying  hours  of  experience  as  a  wingman,  and  having 
demonstrated  solid  flying  skills  and  in-flight  discipline,  pilots  can  then  be  selected  for  FLUG 
training.  Here  they  will  learn  to  lead  an  element  of  two  aircraft  and  eventually  an  entire  flight  of 
four.  Although  the  Air  Combat  Command  (ACC)  (1998)  specifies  general  guidance  for  upgrade 
training,  each  fighter  wing  develops  its  own  program  of  academics,  simulator  exercises,  and 
training  flights.  For  F-16  pilots  these  training  flights  include  a  mix  of  both  air-to-air  and  air-to- 
ground  missions.  Even  though  all  upgrading  pilots  have  gained  considerable  experience  as 
wingmen,  some  missions  are  flown  relatively  infrequently  due  to  airspace  and  resource 
limitations. 

The  4  vs  X  Dissimilar  Air  Combat  Tactics  (DACT)  mission  (four  F-16s  opposed  by  an 
unknown  number  of  threat  aircraft)  is  the  least  practiced  mission  as  it  requires  at  least  eight 
aircraft  (four  friendly  and  four  or  more  enemy)  plus  weapons  controller  support  if  available. 
Even  when  these  resources  are  available,  DACT  training  missions  are  often  constrained  by 
airspace  limitations  that  prevent  teams  from  employing  advanced  tactics.  Similar  resource  limits 
apply  to  Surface  Attack  Tactics  (SAT)  missions  that  incorporate  enemy  fighters  defending 
ground  targets.  As  a  result,  upgrading  pilots  often  find  themselves  learning  to  lead  highly 
demanding  missions  that  they  have  infrequently  practiced  as  wingmen.  If  the  upgrading  pilot 
does  not  demonstrate  the  required  proficiency  during  a  EEUG  mission,  the  mission  must  be 
repeated  until  the  pilot  demonstrates  the  necessary  level  of  expertise.  Depending  on  the  specific 
mission,  this  can  place  significant  demands  on  squadron  resources. 

Objectives 

A  research  program  to  assess  the  effectiveness  of  using  DMT  to  augment  EEUG  training 
was  undertaken  by  AERE/HEA  under  sponsorship  of  ACC’s  DMT  office  and  with  the 
cooperation  of  the  27'*’  Tighter  Wing,  Cannon  AEB,  and  the  552”*^  Air  Control  Wing,  Tinker 
AEB.  The  objective  of  this  program  was  to  determine  how  DMT  experience  incorporating 
exposure  to  numerous  air-to-air  engagements  can  increase  pilot  readiness  for  EEUG  missions 
and  reduce  the  incidence  of  missions  that  were  repeated  due  to  noneffective  upgrading  pilot 
(NEUP)  performance. 

The  research  program  encompassed  four  phases. 
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•  First,  training  records  were  reviewed  to  identify  4-ship  PLUG  missions  that  would  most 
benefit  from  DMT  experience  and  to  establish  baseline  rates  for  sorties  that  were  repeated 
due  to  NEUP  proficiency. 

•  Second,  a  5-day  DMT-FLUG  training  protocol  was  developed  targeting  these  missions. 

•  Third,  DMT-FLUG  training  events  were  conducted  over  a  one-year  period.  During  these 
events,  upgrading  pilots  planned,  briefed,  led,  and  debriefed  missions  of  increasing 
complexity  using  the  Air  Force  Research  Laboratory  (AFRL)  4-ship  DMT  testbed. 

•  Fourth,  transfer  to  aircraft  training  was  assessed  through  review  of  training  records  and 
interviews  with  both  upgrading  pilots  and  their  instructors. 


DMT-FLUG  TRAINING  PROTOCOL 
Review  of  Training  Records 

At  the  27^’’  Fighter  Wing  during  the  review  period,  FLUG  included  14  training  missions, 
10  in  which  the  upgrading  pilot  leads  a  2-ship  element  and  four  leading  a  4-ship  flight.  In  April 
1999,  AFRL  researchers  reviewed  training  records  for  all  pilots  who  had  completed  FLUG  in  the 
previous  three  years  or  were  currently  participating  in  FLUG.  Upgrading  pilot  proficiency  for 
each  training  mission  is  evaluated  by  an  instructor  pilot  who  rates  performance  on  a  number  of 
mission  elements  using  a  scale  ranging  from  1:  Performance  is  safe  but  indicates  a  lack  of 
proficiency,  to  4:  Performance  reflects  an  unusually  high  degree  of  ability.  A  minimum  score  of 
2:  Performance  is  essentially  correct;  recognizes  and  corrects  errors,  on  all  critical  elements  is 
required  for  a  mission  to  be  rated  complete  and  effective.  A  mission  that  is  rated  noneffective 
must  be  repeated  before  an  upgrading  pilot  can  fly  a  checkride  for  FLUG  certification.  While 
researchers  initially  focused  on  overall  mission  scores,  this  proved  to  be  unnecessary  as  nearly 
all  effective  missions  were  rated  2  and  all  noneffective  missions  were  rated  1. 

Training  data  for  over  30  upgrading  pilots  show  that  the  refly  rates  for  noneffective 
performance  ranged  from  0%  for  2-ship  night  intercepts  and  less  than  3%  for  2-ship  basic  fighter 
maneuvers  and  2-ship  basic  surface  attack  to  over  22%  for  2-ship  intercepts.  Among  4-ship 
missions,  the  highest  refly  rates,  15%  were  for  4  vs  X  DACT  and  4-ship  SAT  (see  Figure  1). 
Further,  review  of  individual  scores  and  instructor  comments  reveal  that  noneffective  ratings  for 
the  4-ship  SAT  mission  most  often  resulted  from  poor  reaction  to  surface  and  airborne  threats 
rather  than  for  problems  with  navigation,  air-to-surface  tactics,  or  ordnance  delivery. 

Protocol  Development 

Results  of  the  FLUG  training  data  analysis  were  briefed  to  wing  and  squadron  leaders 
with  the  goal  of  selecting  missions  to  be  complemented  with  DMT.  Wing  leaders  chose  to 
correct  2-ship  mission  problems  using  existing  resources.  Their  most  significant  problem  by  far 
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Mission  description 

Figure  1.  Percentage  of  PLUG  Training  Missions  Repeated  for  Noneffective  Upgrading  Pilot 
Performance  (BFM:  Basic  Fighter  Maneuvers,  SA:  Surface  Attack;  FLUG:  Flight  Lead 
Upgrade;  SAT:  Surface  Attack  Tactics) 

was  the  4  vs  X  DACT  mission  based  on  cost  and  difficulties  of  scheduling  so  many  aircraft  and 
pilots.  The  4-ship  SAT  mission  was  also  selected  for  additional  training  with  DMT  due  to  its 
high  refly  rate.  A  DMT-FLUG  protocol  was  developed  consisting  of  four  missions:  4  vs  4 
Sweep,  4  vs  X  Sweep,  4  vs  X  Defensive  Counterair  (DCA)\  and  SAT  opposed  by  both  airborne 
and  ground  forces.  Within  these  missions,  selected  DMT  FLUG  training  objectives  included 
briefing  and  debriefing  skills,  radar  mechanization;  assessing  the  tactical  situation;  tactical 
execution;  effective  communication;  mutual  support;  and  maintaining  situation  awareness. 
DMT-FLUG  missions  were  designed  to  closely  resemble  FLUG  flying  missions  in  objectives, 
mission  tasks,  and  suggested  briefing  items. 

Design  of  DMT  scenarios  represents  a  middle  ground  between  single-ship  simulator 
training  and  large-force  exercises  (Crane,  1999).  In  single-ship  simulator  training  such  as 
learning  to  respond  to  in-flight  emergencies,  an  instructor  introduces  an  emergency  such  as  an 
engine  malfunction  and  then  waits  for  the  student  to  respond.  Events  are  highly  scripted  and  the 
instructor  can  readily  evaluate  good  vs  poor  performance.  In  contrast,  large-force  exercises  are 
much  less  scripted  at  the  level  of  individual  pilots.  Evaluators  will  know  where  and  when  forces 
will  engage  but  will  have  only  limited  control  over  each  pilot’s  experience.  The  training 
methodology  employed  during  DMT-EEUG  took  advantage  of  the  instructor’s  ability  to  script 
the  actions  of  computer- generated  forces.  The  goal  of  this  system  was  to  create  robust,  focused 


^  Sweep  and  DC  A  missions  are  different  types  of  DACT;  other  DACT  missions  include  Eorce 
Protection  (escort)  and  defending  High-Value  Airborne  Assets  (HVAA). 
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practice  scenarios  that  exercise  pilots’  skills  and  provide  instructors  with  the  tools  required  to 
assess  an  overall  level  of  proficiency.  The  scenarios  varied  in  difficulty  from  a  single  non¬ 
maneuvering  enemy  formation  to  a  continuous  profile  containing  six  maneuvering  formation 
groups  and  16  enemy  aircraft.  Over  60  individual  scenarios  were  available  which  allowed  the 
instructor  to  present  the  PLUG  pilot  with  increasing  challenges  by  changing  the  tactical  variables 
(see  Table  1).  Because  of  the  large  number  of  scenarios  available,  the  tactical  variables  were 
similar  between  each  PLUG  pilot’s  missions,  but  no  specific  scenario  was  ever  repeated  between 
the  pilots.  The  instructor  could  easily  increase  or  decrease  scenario  difficulty  based  on  the  PLUG 
pilot’s  demonstrated  proficiency. 


Table  1.  Variables  Used  in  Programming  DMT-PLUG  Scenarios 


Tactical  Variable 

Values  Used  in  DMT-ELUG 

Number  of  enemy 
aircraft 

4  to  16  in  one  or  more  waves 

Number  of  groups 

1  to  6 

Aircraft  types 

Lighters,  low-level  strikers,  high-fast  flyers 

Rules  of  engagement 

Visual  identification  required  or  beyond  visual 
range 

Enemy  level  of 
awareness 

Unaware  or  aware 

Permanent  kill  removal 

Off  (Shields-up)  or  On  (Shields-down) 

Mission  type 

Offensive  sweep  or  defensive  combat  air  patrol 

Eormations 

Single  heavy  group 

2  groups  divided  in  range 

2  groups  divided  in  azimuth 

2  groups  divided  in  altitude 

3  groups  in  champagne  or  vie  formation 

Lighters  protecting  strikers 

Lighters  plus  high-fast  flyers 

Maneuvers  including  flank,  beam,  drag,  spin, 
and  post-hole. 

Based  on  the  training  objectives  and  logistical  considerations,  a  four  and  one-half  day, 
intensive  protocol  was  developed  with  pilots  flying  two  DMT  missions  per  day  with  each 
mission  consisting  of  several  scenarios  (see  Table  2).  Each  mission  included  a  one-hour 
briefing,  one-hour  simulator  mission,  and  a  one  and  a  half-hour-long  debrief.  Teams  consisting 
of  two  upgrading  PLUG  pilots,  two  wingmen,  an  instructor  pilot,  and  one  or  two  AW ACS 
weapons  controllers  flew  each  mission  type  (i.e.,  4  v  4  Sweep,  4  v  X  Sweep,  DCA,  and  SAT) 
twice.  This  gave  each  PLUG  pilot  the  opportunity  to  brief  and  lead  each  mission  type.  Each 
upgrading  pilot  also  flew  in  the  Number  3  position  leading  the  second  element  within  the  flight 
for  three  missions  and  observed  one  mission  from  the  ALRL  DMT  control  console.  The 
instructor  pilot  flew  four  missions  during  the  week  in  the  Number  2,  3,  or  4  position,  which 
enabled  him  to  assess  the  strengths  and  shortfalls  of  the  simulators  and  allowed  the  other  pilots  to 
observe  one  training  mission  from  the  console.  Approximately  7-8  scenarios  were  flown  during 
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Table  2.  DMT-FLUG  Five-Day  Training  Schedule 


Mon. 

Tue. 

Wed. 

Thu. 

Fri. 

am 

Familiarization 

4  V  4  Sweep 

4  V  X  Sweep 

4  V  X  DCA 

SAT  -  Medium 
and  Low 

Altitude 

pm 

4  V  4  Sweep 

4  V  X  Sweep 

4  V  X  DCA 

SAT  -  Medium 
and  Low 

Altitude 

each  sweep  mission,  3-4  DCA  scenarios,  and  2  scenarios  (one  medium  altitude  and  one  low 
altitude)  were  flown  during  the  SAT  missions.  The  instructor  pilot  flew  four  missions  during  the 
week,  and  observed  the  others  from  the  AFRL  DMT  control  console;  one  of  the  two  wingmen 
observed  from  the  console  when  the  instructor  flew  a  mission.  Characteristics  of  the  missions 
were: 

•  The  4  vs  4  Sweep  mission  consisted  of  scenarios  limited  to  four  enemy  aircraft  in  two 
groups  or  less  (Figure  2).  The  first  several  scenarios  required  the  FLUG  pilots  to  visually 
identify  (VID)  the  target  aircraft  before  they  could  employ  ordnance.  Suggested  briefing 
items  included  VID  mechanics,  sorting  and  targeting  plan,  and  missile  shot/kill  criteria. 
Simulator  “shields”  were  up  which  meant  that  the  F-16  pilots  could  not  be  killed  by  the 
threat  aircraft. 

•  The  4  vs  X  Sweep  mission  limited  the  enemy  to  three  groups  with  no  maneuvering 
restrictions  (Figure  3).  Simulator  shields  were  down  which  allowed  for  permanent  kill 
removal.  High,  fast,  flyer  procedures  and  tactics  were  introduced  as  a  new  briefing  item. 

•  The  4  vs  X  DCA  mission  was  flown  shields-down  and  emphasized  combat  air  patrol 
(CAP)  management  (Figure  4).  Unlike  the  Sweep  scenarios  that  were  terminated  after  one 
force  was  destroyed,  DCA  scenarios  ran  continuously  for  a  15-20  min  vulnerability  period 
during  which  the  F-16  force  was  tasked  to  defend  their  home  station  from  attack.  Enemy 
formations  were  limited  to  three  groups.  However,  the  enemy  groups  had  no  range, 
azimuth,  or  altitude  restrictions  and  a  mix  of  hostile  and  bogey  groups  was  presented.  Up 
to  six  aircraft  engaged  the  F-16s  at  any  time  during  the  scenario  with  up  to  16  total  enemy 
aircraft  in  multiple  waves  over  the  duration  of  the  vulnerability  period.  Pilots  needed  to 
reform  their  CAP  after  each  wave  and  manage  their  fuel  and  missiles.  In  addition,  enemy 
surface-to-air  missiles  were  inserted  along  the  western  edge  of  the  gaming  area.  These 
missiles  would  engage  F-16s  that  strayed  too  far  past  their  assigned  patrol  area. 

•  The  SAT  mission  contained  a  low  and  medium  altitude  profile.  In  addition  to  enemy 
aircraft,  enemy  surface-to-air  missiles  and  antiaircraft  artillery  were  embedded  in  the 
scenario  to  evaluate  threat  awareness  and  reactions  (Figure  5).  The  ingress  route,  target, 
weapons  load,  and  known  enemy  ground  positions  remained  constant  between  the  two 
scenarios;  however,  the  number  and  location  of  enemy  aircraft  varied.  This  mission 
stressed  decision  making  and  air-to-air  tactics  against  enemy  aircraft  given  that  the  F-16’s 
primary  role  was  air  to  ground. 
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Figure  2.  4  vs  4  Sweep  Scenario 


Figure  3.  4  vs  X  Sweep  Scenario 
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Figure  5.  Surface  Attack  Tactics  (SAT)  Scenario 
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DMT-FLUG  TRAINING  EVENTS 


Systems 

DMT-FLUG  training  was  conducted  using  AFRL/HEA’s  DMT  testbed  system,  which 
consists  of  four  interlinked  F-16  simulators  (Figure  6a),  a  eontrol  and  observation  eonsole 
(Figure  6b),  a  constructive  threat  system  providing  semi-autonomous,  computer-generated 
aircraft,  long-distance  network  to  an  AW ACS  simulator  (Figure  6c)  located  at  Brooks  AFB  in 
San  Antonio,  TX,  and  a  mission  replay/debrief  system  (Figure  6d).  For  the  SAT  missions,  the 
Air  Force  Information  Warfare  Center  at  Kelly  AFB,  TX,  provided  a  constructive,  integrated  air 
defense  system.  For  a  more  complete  description  of  AFRL/HEA’s  DMT  testbed,  see  Crane, 
Schiflett,  and  Oser  (2000). 


Eigure  6.  AERE/HEA  DMT  Testbed  Systems:  a)  E-16  Simulator,  b)  Control  and  Observation 
Console,  e)  AW  ACS  Simulator,  and  d)  Replay/Debrief  System 
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Schedule 


DMT-FLUG  training  events  were  conducted  in  June,  August,  and  October  1999,  and 
February  2000.  Participants  were  selected  by  their  squadrons  based  on  availability  and  status 
within  the  FLUG  program.  Ideally,  the  upgrading  pilots  had  completed  their  2-ship  FLUG 
training  flights  and  were  beginning  their  4- ship  flights.  Due  to  deployments  and  other 
commitments,  pilots  in  all  stages  of  FLUG  training  participated.  Weapons  controllers  from  the 
552”^*  Air  Control  Wing  and  the  970'^  Airborne  Air  Control  Squadron  were  assigned  by  their 
squadrons  based  on  availability.  All  pilots  and  weapons  controllers  were  combat  mission  ready. 

Participants 

Twelve  upgrading  pilots  with  200  to  485  flight  hours  (median  =  410)  in  the  F-16 
participated  in  DMT-FLUG  together  with  6  instructor  pilots,  9  wingmen,  and  10  weapons 
controllers.  None  of  the  upgrading  pilots  had  any  previous  experience  in  other  fighter  aircraft. 
One  pilot  was  called  away  due  to  a  family  emergency  after  one  DMT-FLUG  mission;  however, 
one  of  the  wingmen  was  also  in  FLUG  training  and  replaced  him  in  the  sample. 

RESULTS 

Three  types  of  data  were  collected  to  assess  the  effect  of  DMT  on  FLUG  training: 
performance  in  subsequent  aircraft  training  events,  performance  during  DMT-FLUG  missions, 
and  feedback  from  participants. 

Performance  in  Subsequent  Aircraft  Training 

As  of  1  September  2000,  8  of  the  12  upgrading  pilots  who  participated  in  DMT-FLUG 
have  successfully  completed  the  FLUG  program  without  any  repeated  rides.  Two  pilots, 
however,  were  accepted  into  the  B-2  program,  and  their  squadron  waived  the  requirement  for  a  4 
V  X  DACT  ride  for  these  pilots.  Both  successfully  completed  a  FLUG  checkride,  which  is  a  4- 
ship  SAT  mission.  Eight  pilots  successfully  completed  the  4  v  X  DACT  ride  on  the  first  attempt. 
One  DMT-FLUG  pilot  was  required  to  repeat  his  4  vs  X  DACT  mission  for  noneffective 
performance  on  his  mission  briefing.  This  pilot  participated  in  DMT  FLUG  in  October  1999  and 
did  not  fly  the  DACT  mission  until  April  2000.  In  addition,  this  pilot  flew  his  DACT  mission  as 
part  of  an  exercise  consisting  of  several  4-ship  air-to-air  missions  so  that  scheduling  an 
additional  4  vs  X  sortie  was  not  required.  The  remaining  pilot  was  still  in  FLUG. 

Performance  in  DMT - FLUG 

Each  scenario  flown  during  a  DMT-EEUG  simulator  period  was  graded  by  a  laboratory 
subject-matter  expert.  Initially  the  rating  scales  and  mission  tasks  were  similar  to  their  mission 
grade  sheets  used  at  home  station.  However,  early  data  analyses  identified  several  mission  tasks 
that  showed  little  variability  over  the  week  and  some  mission  tasks  that  were  rarely  observed 
during  DMT-EEUG  missions.  To  overcome  these  difficulties,  the  number  of  graded  tasks  was 
reduced  and  a  new  scale  was  devised  to  help  obtain  more  meaningful  information.  The  new  3- 
level  rating  scale  (-,  0,  -i-)  asked  the  subject-matter  expert  to  evaluate  mission  performance  as 
being  either  below  average  for  an  upgrading  EEUG  pilot,  average,  or  above  average.  Coincident 
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with  the  new  rating  scale,  benchmark  scenarios  were  introduced  to  better  evaluate  pilot 
performance  during  the  week.  Two  sets  of  benchmark  scenarios  with  equal  difficulty  were 
selected.  Benchmark  scenarios  used  with  upgrading  pilots  on  the  same  team  were  essentially 
mirror  images.  The  first  benchmark  scenario  was  administered  during  the  4  vs  4  Sweep  mission, 
and  the  second  was  given  during  the  4  vs  X  DCA  mission.  Six  pilots  have  been  evaluated  using 
the  new  scale  and  benchmark  scenarios.  The  overall  performance  between  their  benchmark 
scenarios  improved  for  all  six  PLUG  pilots.  Several  individual  mission  tasks  also  showed 
noteworthy  improvement  in  half  or  more  of  the  comparisons.  These  areas  included  mutual 
support,  tactical  decision  making,  correct  picture  call,  work  with  AW ACS  weapons  controllers, 
tactical  execution,  radar  mechanization,  and  targeting/sorting  all  bandits  (see  Figure  7). 


Mission  Task 

Figure  7.  Changes  in  Mean  Mission  Performance  Ratings  Between  the  First  and  Second 
Benchmark  Missions  During  DMT-FLUG  Training 

Increases  in  performance  scores  between  the  first  DACT  mission  (4  vs  4)  and  the  third 
(DCA)  were  not  uniform  across  skills.  Scores  for  individual  skills  increased  consistently  from 
the  first  mission  to  the  third.  An  individual  radar  skill,  detect  all  targets  (Figure  8a),  and  an 
individual  communications  skill,  correct  picture  call  (Figure  8b)  are  examples.  Improvements  in 
scores  for  team  skills,  however,  required  more  DMT  experience.  A  team  radar  skill,  target  and 
sort  all  bandits  (Figure  8a),  and  a  team  communications  skill,  work  with  AWACS  weapons 
controller  (Figure  8b),  decreased  as  mission  complexity  increased  from  4  vs  4  to  4  vs  X.  With 
additional  experience  in  complex  scenarios,  however,  performance  scores  increased. 


11 


Individual  Skill:  Detect  All  Groups 
— B —  Team  Skill:  Target  and  Sort  All  Bandits 


Individual  Skill:  Picture  calls 
—a — Team  Skill:  Work  w/  AWACS 


Figure  8.  Changes  in  Mean  Performance  Scores  for  Individual  and  Team  Skills: 
a)  radar  skills,  b)  communication  skills 


The  number  of  times  that  a  flight  member  was  killed  by  enemy  action  increased  slightly 
between  the  first  and  last  air-to-air  missions;  however,  this  change  was  small  compared  to  the 
increased  complexity  of  the  scenarios  and  the  number  of  enemy  aircraft.  The  number  of  total 
fratricide  incidents  was  reduced  from  six  during  the  4  vs  4  missions,  to  only  one  incident  during 
the  4  vs  X  DC  A  missions. 

Pilot  Feedback 

Pilots  and  weapons  controllers  were  asked  to  provide  feedback  to  the  research  team  after 
each  mission  and  at  the  end  of  the  week.  Comments  fell  into  three  categories:  system  problems, 
training  effectiveness,  and  recommendations. 

Comments  regarding  system  problems  included  minor  issues  that  were  typically 
corrected  within  a  simulator  session  and  more  significant  problems  with  the  testbed.  Minor 
problems  include  head- tracker  failures,  radio  communications  not  correctly  recorded  for  debrief, 
too  much/too  little  air  conditioning  in  cockpit,  and  visual  system  failure  requiring  restart.  More 
significant  comments  include  inability  to  determine  aspect  of  another  aircraft  beyond  4,000  feet 
(see  Crane,  Schiflett  &  Oser,  2000),  and  modeling  of  threats  and  countermeasures. 

Pilots  and  weapons  controllers  participating  in  the  DMT-FLUG  program  agreed  that 
DMT  provides  effective  training  for  multiship,  multibandit  air  combat.  Strengths  of  the  system 
are  the  opportunity  for  multiple  engagements  within  an  hour  and  the  high  level  of  situation 
awareness  provided  by  the  replay  system  during  debrief.  Specific  skills  cited  include  radar 
mechanization,  work  with  weapons  controllers,  communication,  tactical  execution,  situation 
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awareness,  briefing,  and  confidence  building.  Pilots  also  agreed  that  within- visual-range  air 
combat  skills  and  aircraft  handling  are  not  well  trained  in  the  testbed  DMT  system. 

In  addition  to  correcting  system  problems,  participants  offered  several  recommendations 
for  improving  DMT  overall  and  DMT-FLUG.  There  was  disagreement  whether  an  intensive 
syllabus  lasting  several  days  such  as  DMT-FLUG  was  more  effective  than  the  more  traditional 
simulator  mission  first  followed  by  the  aircraft  mission  approach.  While  some  pilots  wanted  to 
rehearse  each  mission  before  flying  it,  all  agreed  that  there  is  considerable  benefit  to  an  intensive 
DMT  air-combat  program.  One  instructor  pilot  stated  that,  “Pilots  overcome  simisms  first,  then 
demonstrate  initial  proficiency,  and  there  is  exponential  improvement  after  that.”  A  second 
recommendation  was  to  integrate  the  weapons  controller  into  the  brief  and  debrief  rather  than  use 
telephone  brief/debrief  between  training  sessions.  Suggestions  included  video  teleconference 
and  interactive  whiteboards. 

Several  weeks  after  returning  to  Cannon  AFB,  questionnaires  were  sent  to  pilots 
regarding  the  effectiveness  of  DMT.  Responses  have  been  received  from  9  of  the  17  pilots  who 
participated  in  DMT-FLUG:  5  upgrading  pilots,  2  instructor  pilots,  and  2  wingmen.  Four  of  the 
five  upgrading  pilots  evaluated  DMT-FLUG  experience  as  significantly  transferring  to  the 
aircraft;  the  fifth  pilot  rated  transfer  as  moderate.  Specific  skills  that  were  cited  as  showing 
transfer  were  briefing,  communications,  beyond-visual-range  tactics,  radar  mechanization, 
situation  awareness,  decision  making,  leadership,  plus  developing  and  using  backup  / 
contingency  plans.  One  of  the  upgrading  pilots  expressed  concern  that  DMT  is  too  easy  and 
lacks  distracting  elements  of  actual  flying  such  as  pulling  g’s,  noise,  and  vibration.  One  IP  rated 
DMT-FLUG  as  transferring  significantly  to  the  aircraft  citing  pacing  and  situation  awareness  as 
skills  well  trained  in  DMT.  The  other  IP,  however,  stated  that  there  was  no  transfer  and  that 
DMT  is  no  more  than  an  expensive  video  game  that  does  not  provide  useful  training.  The  two 
wingmen  stated  that  DMT  transferred  moderately  to  the  aircraft  with  the  most  positive  transfer 
for  situation  awareness,  radar  mechanization,  and  4-ship  tactics.  All  pilots  agreed  that  the  4  vs  X 
DCA  mission  was  the  most  effective  and  that  the  SAT  mission  was  least  effective.  Pilots  were 
asked  whether  they  would  recommend  that  DMT  be  used  in  an  intensive  syllabus  as  in 
DMT-FLUG  or  a  more  traditional  approach  of  alternating  simulator  and  aircraft  training 
missions.  Five  of  the  nine  pilots  preferred  the  traditional  approach  so  that  DMT  missions  could 
be  tailored  to  specific  aircraft  training  flights  although  one  pilot  expressed  concern  that  this 
would  slow  the  overall  pace  of  training.  Two  pilots  preferred  the  intensive  approach  and  two  did 
not  respond.  Recommendations  for  how  frequently  pilots  should  use  DMT  ranged  from  “never” 
to  “any  day  that  I’m  not  flying.” 


DISCUSSION 

As  applied  to  FLUG  training,  results  to  date  demonstrate  that  experience  in  DMT 
enhances  leadership  skills  for  multiship,  multibandit  air  combat.  This  is  most  likely  a  function  of 
both  DMT  system  capabilities  and  constraints  on  pilots’  ability  to  train  in  the  aircraft  that  limit 
opportunities  for  gaining  experience  over  a  wide  range  of  tactical  situations.  Observations  of 
pilots  during  DMT-FLUG  exercises  demonstrated  that  increases  in  mission  performance  resulted 
from  both  enhanced  skills  and  from  opportunities  to  review  missions  and  to  assess  the  results  of 
individual  actions.  The  following  examples  illustrated  these  effects: 
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•  One  upgrading  pilot  flying  the  4  vs  4  scenario  depicted  in  Figure  2  had  briefed  that  for  this 
situation  (enemy  aircraft  groups  separated  in  azimuth),  the  F-16s  would  divide  into  two 
elements  and  bracket  to  the  outside  of  the  enemy  groups  with  pilots  flying  in  the  numbers 
1  and  3  positions  shooting  the  far  north  and  south  bandits  respectively.  The  upgrading 
flight  leader  also  briefed  that  F-16  pilots  were  to  fire  radar  guided  missiles  and  then  turn 
away  before  entering  enemy  missile  range,  Figure  9a.  The  upgrading  pilot,  however, 
lacked  the  skills  required  to  lead  the  flight,  select  the  appropriate  tactical  plan, 
communicate  on  the  radio,  and  use  the  radar  to  sort  the  formation  before  turning  away. 
The  upgrading  pilot  fired  unsorted  into  the  bandit  formation  and  turned  away  while  the 
wingman  correctly  sorted  his  target.  Both  missiles  ended  up  killing  the  same  aircraft  and 
the  F-16  pilots  were  required  to  expend  time  and  fuel  defending  themselves  from  the 
surviving  MiG,  Figure  9b.  After  additional  practice,  the  upgrading  pilot  improved  his 
leadership  and  radar  mechanization  skills  and  was  able  to  focus  on  more  complex  team 
.skills. 


Figure  9.  Briefed  Plan  Compared  to  Actual  Execution  of  a  4  v  4  Sweep  Mission 
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•  Another  upgrading  pilot  was  flying  as  Number  3  on  a  SAT  mission  leading  the  second 
element  within  the  flight  (see  Fig.  5).  The  F-16s  successfully  defended  themselves  against 
several  MiG-29s  and  dropped  their  bombs  on  target.  As  the  F-16s  were  egressing  the 
target  area,  a  single  MiG  was  detected  by  AW ACS  many  miles  behind  them  (Fig.  10a). 
The  upgrading  pilot  turned  away  from  the  formation  supported  by  his  wingman  and  shot 
down  the  MiG  (Fig.  10b).  When  they  turned  back  toward  friendly  airspace,  however,  they 
were  engaged  by  a  previously  briefed  surface-to-air  missile  and  both  killed  (Figure  10c). 
During  debrief,  it  was  clear  that  the  MiG  was  not  a  threat  and  the  F-16s  could  have  flown 
home  safely.  By  diverting  to  engage  the  MiG,  the  upgrading  pilot  pulled  the  formation  off 
course  allowing  his  element  to  be  engaged  by  the  surface-to-air  missile.  While  the  pilot’s 
air  combat  skills  were  excellent,  his  decision  to  engage  was  a  serious  error. 


a)  Tactical  situation  as  F-16s  eggressed  target  area;  a  single  MiG-29  is 
detected  by  AW  ACS 


b)  Upgrading  PLUG  pilot  flying  as  second 

element  lead  turns  to  attack  MiG-29  surface-to-air  missile  after  failing  to 

return  to  planned  course 


Figure  10.  Example  of  poor  situation  awareness  and  decision  making  during  SAT  mission 
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One  of  the  objectives  of  DMT-FLUG  research  was  to  identify  the  ways  in  which  DMT 
experience  could  increase  pilot  readiness  for  PLUG  missions.  These  examples  illustrate  two 
training  benefits  of  DMT.  The  first  is  providing  increased  opportunities  to  train  individual  .skills 
that  must  take  place  in  a  multiplayer  context.  Radar  sorting  and  targeting  requires  multiple 
friendly  and  enemy  aircraft;  opportunities  to  practice  this  skill  in  the  air  are  highly  limited.  DMT 
provides  many  opportunities  to  practice  this  skill  in  a  naturalistic  context  and  within  a  short 
period  of  time.  This  observation  is  in  agreement  with  Bell  (1999)  and  Schneider  (1989)  who 
propose  that  multiple  opportunities  to  practice  a  skill  in  a  variety  of  contexts  will  result  in 
substantial  performance  enhancement.  A  second  training  benefit  from  DMT  is  the  opportunity  to 
gain  experience  in  making  decisions.  Klein  (2000)  asserts  that  lack  of  experience  is  the  primary 
reason  for  poor  decision  making.  Without  high  levels  of  experience,  decision  makers  often  fail 
to  recognize  the  implications  of  various  situations  and  select  inappropriate  courses  of  action. 
Experience  in  DMT  can  augment  aircraft  experience  by  providing  pilots  with  multiple 
opportunities  to  execute  various  tactics  and  to  review  the  results  of  their  actions. 

In  the  next  iteration  of  the  DMT-FLUG  training  effort,  we  will  be  developing  more 
comprehensive  training  and  measurement  capabilities  that  will  permit  us  to  identify  which 
behaviors  improve  and  how  DMT  helped  in  the  improvement  process.  Our  findings  to  date 
indicate  that  minimum  capabilities  for  effective  DMT  include  the  following: 

•  DMT  systems  must  be  supported  by  programmable  scenario  generation  tools  that  permit 
the  incorporation  of  instructional  principles  and  training  strategies  to  foster  skill 
development  and  retention. 

•  Mission  replay  and  after-action  reviewing  capability  are  necessary  capabilities  for  an 
instructionally  viable  training  and  rehearsal  system. 

•  A  construct-oriented  individual,  team,  and  mission  performance  measurement  system. 

Simple  practice  or  free-play  are  not  efficient  training  strategies.  Programmable  scenario 
generation  tools  including  computer-generated  threats  with  the  capability  for  autonomous  action 
permit  instructors  to  take  advantage  of  findings  from  research  on  instructional  principles  to 
design  training  and  rehearsal  events  that  will  meet  specified  objectives.  In  multiship  air  combat, 
training  objectives  for  upgrading  pilots  include  learning  to  recognize  enemy  aircraft  formations 
and  selecting  an  appropriate  tactic,  communicating  the  plan  to  the  rest  of  the  flight,  executing  the 
plan,  and  changing  tactics  as  required.  The  programmable  scenario  generation  tools  in  DMT- 
FLUG  were  used  to  create  enemy  formations  and  tactics  that  varied  across  a  number  of 
dimensions  as  summarized  in  Table  1.  This  range  of  scenarios  combined  with  the  opportunity 
for  teams  to  fly  several  scenarios  in  an  hour  provide  upgrading  pilots  with  opportunities  to  gain 
experience  with  many  more  tactics  and  situations  than  would  be  possible  in  the  aircraft. 

Another  key  element  in  the  success  of  DMT  is  the  replay  and  debrief  facility.  In  the 
AFRL  testbed,  each  cockpit’s  radar,  radar  warning  receiver,  head-up  display,  and  stores 
management  system  are  recorded  and  played  back  synchronized  with  a  plan  view  display  and  all 
radio  communications.  The  flight  leader  can  pause,  rewind,  or  zoom  the  display  as  required. 
Using  the  replay  system,  pilots  can  review  the  information  that  was  available  to  them  inside  the 
cockpit  together  with  the  plan  view  display’s  depiction  of  the  complete  tactical  situation. 
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Execution  errors,  poor  communication,  and  unplanned  contingencies  are  quickly  apparent. 
Debriefs  were  limited  to  90  minutes  for  scheduling  purposes,  which  forced  teams  to  focus  on 
high-level  skills  such  as  communication  and  execution  without  getting  absorbed  into  analysis  of 
individual,  procedural  skills. 

Finally,  it  is  important  to  note  that  the  results  presented  in  this  report  represent  an 
evolution  of  our  training  paradigm  from  one  of  structured  practice  to  one  that  is  more  focused  on 
both  objectives-driven  instruction  and  construct-oriented  performance  measurement.  The 
lessons  learned  from  this  initial  DMT-FLUG  effort  have  provided  considerable  insight  into  the 
problems  and  needed  capabilities  associated  with  implementing  a  high-fidelity,  adaptive  training 
and  rehearsal  system  for  Air  Force  pilots.  The  following  summary  describes  future  directions  in 
research  being  undertaken  at  AFRF/HEA  based  on  the  results  of  DMT-FFUG. 

Training  Needs  Assessment  and  Content  Validation 

In  continuing  DMT-FFUG  research,  we  will  be  conducting  systematic  assessment  of 
training  needs  at  other  F-16  bases.  The  goal  of  the  needs  assessment  is  to  elaborate  the  specific 
tasks  and  mission  areas  that  are  problematic  for  pilots  in  upgrade  training.  Additionally,  the 
needs  assessment  will  provide  the  necessary  data  to  develop  an  instructionally  principled 
curriculum.  By  focusing  on  key  needs,  we  can  design  a  syllabus  that  fosters  the  development 
and  retention  of  skills  needed  for  FFUG  success.  Moreover,  the  needs  assessment  will 
complement  our  current  research  activities  in  the  definition  of  mission-essential  competencies, 
i.e.,  the  knowledge,  skills,  attitudes,  abilities,  and  capabilities  required  for  combat  mission 
readiness.  This  effort  will  help  us  develop  a  more  comprehensive,  objectives -oriented  syllabus 
for  DMT-FFUG  and  identify  the  important  performance  parameters  associated  with  mission 
effectiveness.  It  will  also  help  us  in  the  development  of  metrics  to  quantify  training  impact  in 
terms  of  combat  mission  readiness  and  performance. 

Development  of  Performance  Criteria  and  Measurement  Instruments 

We  are  currently  developing  and  testing  evaluation  instruments  for  individual  and  team 
effectiveness  based  on  mission-essential  competencies.  By  adopting  a  competency-based 
approach  to  training  development  and  evaluation,  we  are  developing  data  collection  and  analysis 
methods  for  use  across  the  entire  spectrum  of  DMT  activities  (e.g.,  mission  preparation,  briefing, 
execution,  and  debriefing).  Measurement  techniques  are  being  developed  for  analysis  of  verbal 
communications,  pilots’  mental  models,  team  processes,  and  mission  outcomes. 

•  Content-analysis  methods.  Content-analysis  instruments  are  being  developed  to  evaluate 
verbal  communications  to  better  understand  the  quality  of  briefings,  within-mission 
communication,  and  debriefings.  In  addition,  measures  of  attitudes  toward  training, 
motivation,  and  perceived  application  and  utility  of  the  training  have  been  developed  for 
use  in  other  training  research  and  will  be  extended  to  FFUG. 

•  Mental  modeling  assessments.  We  are  also  conducting  a  series  of  studies  on  structured 
assessments  of  learning  such  as  situational  judgment  assessments,  mental  modeling. 
Pathfinder,  concept  mapping,  and  multidimensional  scaling.  These  are  being  applied  to 
assess  learning  and  understanding  at  both  individual  and  team  levels. 
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•  Process  and  outcome-oriented  measurement.  Using  mission-essential  competency 
definitions,  it  is  possible  to  identify  process-oriented  behaviors  that  are  associated  with 
effective  mission  performance.  We  will  then  be  able  to  develop  process-oriented 
measures  that  will  help  us  quantify  the  effects  of  DMT  experience  on  individual  and  team 
knowledge  acquisition,  crew  coordination/flight  integrity  (including  leadership  and 
assertiveness),  situational  awareness,  communication,  decision  making/risk  management, 
task  management,  and  mission  planning/debrief.  As  a  part  of  these  activities,  we  will 
develop,  test,  and  validate  protocols  to  capture  data  related  to  the  identified  processes.  The 
key  process  behaviors  and  metrics  will  be  used  to  further  refine  training  objectives  and  to 
identify  opportunities  to  improve  training  content  and  training  policy. 

Finally,  we  are  evaluating  the  reliability  and  validity  of  mission  outcome  measures 
including  exchange  ratios  and  missile  effectiveness.  Based  on  this  evaluation,  we  will  be 
refining,  automating,  and  applying  measures  for  simulator  and  flying  performance.  It  is  critical 
that  any  developed  measurement  system  provide  reliable  and  valid  data  to  demonstrate  that 
training  has  had  an  impact  on  actual  aircrew  competencies,  attitudes  toward  training,  motivation 
to  train,  expectations  regarding  training  and  transfer,  knowledge  mastery  and  subsequent  job 
performance.  Furthermore,  the  data  and  instrumentation  will  serve  both  a  formative  assessment 
function  by  providing  feedback  on  problem  areas  in  the  training  and  a  summative  function  by 
permitting  cost  and  mission  impact  assessments. 

SUMMARY 

DMT-FLUG  research  has  demonstrated  both  the  utility  and  effectiveness  of  DMT  to 
complement  aircraft  training.  DMT  supports  upgrade  training  by  providing  opportunities  for 
pilots  to  plan,  brief,  fly,  and  debrief  many  more  air-to-air  missions  than  currently  possible  in 
aircraft.  In  addition,  DMT  provides  instructors  the  capability  to  choose  scenarios  that  exercise 
selected  skills.  Subsequent  DMT-FLUG  research  will  focus  on  developing  a  comprehensive 
training  syllabus  that  will  incorporate  needs  analysis,  specification  of  essential  competencies, 
scenario  development  and  selection  based  on  principles  of  instruction,  and  performance 
measurement  systems  that  will  assess  both  mission  outcome  and  team  processes. 
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